Are the Right People Measuring the Right Things ?

T he effectiveness of the CMMI framework at helping organizations achieve business objectives is currently a hotly debated topic [1]. One of my clients responded to my recent suggestion to consider using the CMMI and Agile techniques together by stating: We do not place much value in the CMMI because we have not seen a difference between Level 4 and 5 organizations and those rated lower. At a recent conference in a presentation about high process-maturity project failures, we also heard some failures were due to non-process effects such as people [2]. To help us understand what might be going on, I would like to share a case study. I have a client who has been using the CMM/CMMI framework for many years to aid process improvement. According to traditional literature, they appear to be doing the right things. They have discussed and captured their business objectives using the Goal-Question-Metric technique, and they have aligned their measures with these objectives. They have standardized their processes and trained their people [3]. This client has multiple product lines. Their processes and training include a strong emphasis on product baseline management with disciplined change approvals. Their established and documented business objectives include increasing off-the-shelf product sales, reducing unique customer customization, and meeting cost and schedule commitments. To help achieve these objectives, the client has been striving to increase management visibility of work through the use of standard product and process metrics. This all seems to make sense. So a natural question arises: Are they achieving their business objectives? While isolated success stories exist, most senior managers – as well as the company metrics – indicate the organization has fallen far short of its goals. While the organization continues to propose and bid off-the-shelf solutions, standard metrics show significant unplanned product changes and cost and schedule overruns. A frequent refrain heard at senior management reviews is the following: " Why are we making all these unplanned and un-bid changes? " So what happened? While this organization appears to have followed the prevailing wisdom in deploying sound processes, why have they failed to achieve their objectives? This case is not unique. Over the past 10 years, I have observed variants of this pattern in multiple organizations. A closer look at this case may provide insight that can help answer why many organizations seeking higher process maturities are failing to enjoy the promised results of their …